Method and apparatus for motion estimation using block matching

ABSTRACT

Block matching is well known as a robust and intuitively simple method of motion estimation for television pictures. However, when the purpose of the motion estimation is to provide accurate interpolation between images, conventional block matching suffers from the problem that the blocks are situated in the original fields rather than in the fields to be interpolated, leading to possible errors in the calculated motion field. The invention uses two-sided block matching. The search window is shared between the two input fields so that the candidate motion vectors related to be candidate pixel blocks all pass through the same points in the field to be interpolated. Thus, the &#34;current block&#34; becomes a notional area in the field to be interpolated, and each of the two search windows extends to half the maximum motion vector in each direction. Each motion vector points forward to the forward field by half its value and backward to the backward field by half its value. Two-sided block matching solves the positional error problem because the current blocks cover the interpolated field and are correctly placed in the interpolated field, and it solves the problem of gaps and conflicts because every point in the interpolated field has one motion vector assigned to it.

The present invention relates to a method and to an apparatus for motionestimation using block matching.

BACKGROUND

Block matching is well known as a robust and intuitively simple methodof motion estimation for television pictures.

FIG. 7 shows a possible architecture for a one-sided block matchingmotion estimator with a range of ±2 pixels per field horizontally andvertically, to integer accuracy. This architecture is based onprocessing elements, one per candidate motion vector. Each processingelement PE accumulates the errors between the pixels (which form a grouprelated to a candidate block position and vector, respectively) arrivingat its two inputs, storing partial results in order to arrive at a totalerror for each block. The processing elements are connected together bymeans not shown in the figure in order to find which one yields theminimum error for each block. The processing elements may calculate theminimum absolute value or the minimum squared error.

In the one-sided motion estimator in FIG. 7, all the processing elementsPE have a common input INP, corresponding to the current block. Therelative displacements between the pixels in the search window areprovided by a network of line LD and sample SD delays that follow thefield delay FD connected to input INP.

U.S. Pat. No. 5,162,907 shows one-sided block matching, in which thereference picture is not the picture to be interpolated. However, whenthe purpose of the motion estimation is to provide accurateinterpolation between images, for example in a 50 Hz to 100 Hzupconverter, conventional block matching suffers from the problem thatthe blocks are situated in the original fields rather than in the fieldsto be interpolated, leading to possible errors in the calculated motionfield.

SUMMARY

It is one object of the invention to disclose a method of blockmatching, which overcomes this problem.

It is a further object of the invention to disclose an apparatus whichutilizes the inventive method.

The invention uses two-sided block matching. The advantages of two-sidedblock matching are not confined to 50 Hz to 100 Hz upconversion butapply to any temporal interpolation, for example 50 Hz to 60 Hz or 50 Hzto 75 Hz standards conversion, for which motion vectors are required atpoints in time between input fields.

In a conventional, or `one-sided`, block matching, one possiblearchitecture in FIG. 1, the current field (or frame) CF of the pictureis divided into blocks, typically rectangular. For each current blockCB, a search is made among overlapping blocks of the same size in theprevious field (or frame) PF to find one that matches the current-fieldblock CB best, usually using a mean-square or mean-absolute errorcriterion summed across the pixels in the block. The relative positionof the best-matching block in the previous frame gives the chosendisplacement, or motion, vector MV for the current block. For a givencurrent block CB, the set of pixels involved in the previous frame isknown as the search window SW corresponding to that block. Thus, theblock matching process makes use of the current block and the searchwindow.

The problem with one-sided block matching, when applied to field rateupconversion, is that the blocks for which motion vectors are sought liein the input fields rather than in the fields IF to be interpolated.This leads to a positional error PE (with respect to the position PCB ofthe current block) in FIG. 2 in the information used to estimate themotion vector, which increases with the speed of motion. This errorbecomes more important when block matching is applied to localizing. theboundaries between moving areas, as described in EP93 402188 of theapplicant.

The positional error PE can be corrected by displacing the block in theinterpolated field IF appropriately according to the motion vectors, butthis leads to the occurrence of gaps GP (areas in the interpolated fieldfor which no motion vectors exist) and, equally, of areas CT for whichtwo or more motion vectors conflict, as shown in FIG. 3.

The inventive solution to the above problem is to use two-sided blockmatching which is explained with reference to FIG. 4. Here, the searchwindow (BSW and FSW) is shared between the two input fields BF and FF insuch a way that the candidate motion vectors (related to the candidatepixel blocks CPBB and CPBF) all pass through the same points in thefield IF to be interpolated. Thus, the `current block` becomes anotional area in the field to be interpolated, and each of the two newsearch windows BSW and FSW extends to half the maximum motion vector ineach direction. Each motion vector points forward (forward half ofmotion vector FHMV) to the forward field FF by half its value andbackward (backward half of motion vector BHMV) to the backward field BFby half its value. Both vector parts depend on the same candidate motionvector.

Two-sided block matching solves the positional error problem because thecurrent blocks IFB cover the interpolated field and are correctly placedin the interpolated field, and it solves the problem of gaps andconflicts because every point in the interpolated field has preciselyone motion vector assigned to it, see FIG. 5.

One problem with two-sided block matching is that, for a given motionvector accuracy, pixel coordinates need to be expressed twice asaccurately as for one-sided block matching. This is particularlydifficult if the motion vectors are required to integer accuracy,because the forward-and backward-pointing halves of the motion vectorwill then be expressed to the nearest half-integer. This problem can beovercome either

(i) by interpolating the field to provide the half-pixel points (forwhich simple bilinear interpolation will suffice) or

(ii) by rounding the forward and backward half-vectors in such a waythat their sum remains correct.

Method (i) involves an increase in hardware complexity, while method(ii) avoids the need for half-pixel interpolation but re-introduces aslight positional error (i.e. the vector is not calculated at the exactposition, but because the present vector position is very near to therequired one the direction of the ideal vector will be very similar tothe direction of the calculated one) of up to half a pixel in theinformation used to estimate the motion vectors, an error which has beenshown to have negligible effect on the final picture quality. These twomethods are illustrated (for one dimension only) in FIG. 6. FIG. 6Adepicts the two-sided case with interpolated pixels IP for method (i)and FIG. 6B the two-sided case with slight positional error SPE (maximumcase) for method (ii).

In principle, the inventive method is suited for motion estimation usingblock matching within a preselected search window width, wherein eachsuccessive pixel block of a current field or frame is matched withrespective candidate pixel blocks within a search window of a backwardfield or frame having a half of said preselected width and with thecorresponding pixel blocks within the corresponding search window of aforward field or frame having a half of said preselected width in orderto select a motion vector related to the location of the best matchingblock of the backward field or frame and to the location of the bestmatching block of the forward field or frame, whereby the currentposition of said search window is related to the position of the currentpixel block of said current field or frame.

In principle the inventive apparatus is suited for motion estimationusing block matching within a preselected search window width andincludes:

field or frame delay means and line delay means and sample delay meansproviding for each successive pixel block of a current field or framethe pixel positions of a search window of a backward field or framehaving a half of said preselected width and the pixel positions of thecorresponding search window of a forward field or frame having a half ofsaid preselected width;

processing elements arranged in respective order with said delay means,for matching each of said successive pixel blocks of a current field orframe with the corresponding candidate pixel blocks within said searchwindows by accumulating the errors between such pixels which arrive atits two inputs, storing partial results in order to arrive at a totalerror for each block;

detection means connected to said processing means for selecting amotion vector related to the location of the best matching pair offorward and backward blocks, a first part of which is related to thebackward field or frame and a second part of which is related to theforward field or frame, whereby the current position of said searchwindows is related to the position of the current pixel block of saidcurrent field or frame.

DRAWINGS

Preferred embodiments of the invention are described with reference tothe accompanying drawings, which show in:

FIG. 1 one-sided block matching;

FIG. 2 positional error in one-sided block matching;

FIG. 3 motion vector gaps and conflicts in one-sided block matching;

FIG. 4 two-sided block matching;

FIG. 5 avoidance of gaps and conflicts in two-sided block matching;

FIG. 6 overcoming the accuracy problem;

FIG. 7 one architecture for one-sided block matching;

FIG. 8 one architecture for two-sided block matching.

PREFERRED EMBODIMENTS

FIG. 8 shows a possible architecture for a two-sided block matchingmotion estimator with a range of ±2 pixels per field horizontally andvertically, to integer accuracy. This architecture is based onprocessing elements, one per candidate motion vector. Each processingelement PE accumulates the errors between the pixels (which form a grouprelated to a candidate block position and vector, respectively) arrivingat its two inputs, storing partial results in order to arrive at a totalerror for each block. The processing elements are connected together bymeans not shown in the figure in order to find which one yields theminimum error for each block.

The processing elements may calculate the minimum absolute value or theminimum squared error.

The two-sided motion estimator in FIG. 8 uses method (ii) describedabove to overcome the accuracy problem. The line LD and SD sample delaysused to generate the relative search window displacements are now sharedbetween the two fields, and both inputs of each processing element PEdepend on the motion vector being tested, whereby the one input includesthe field delay FD connected to input INP.

Advantageously, with the architecture described, the two-sidedimplementation is actually slightly simpler than the one-sided of FIG.7.

The inventive motion compensation improves the subjective picturequality and can also be used in slow motion generation, noise reduction,standard conversion and for controlling LCD displays.

We claim:
 1. Apparatus for motion estimation using two-sided blockmatching wherein a current field or frame to be interpolated and whichis non-existing in an original picture signal is regarded as havingnotional pixel blocks which cover the field or frame to be interpolatedand are correctly placed in that field or frame to be interpolated,comprising:field or frame delay means, line delay means and sample delaymeans for providing for each current of said notional pixel blockspixels of the required pixel positions of a forward search window,having a preselected width, of a forward field or frame, further fieldor frame delay means, line delay means and sample delay means forproviding for the current of said notional pixel blocks, the pixels ofthe required pixel positions of a backward search window, having saidpreselected width of a backward field or frame; processing elements eachhaving two inputs and being connected to said field or frame delaymeans, line delay means and sample delay means and/or to said furtherfield or frame delay means, line delay means and sample delay means formatching respective forward candidate pixel blocks within the forwardsearch window in the forward field or frame with respective backwardcandidate pixel blocks within the backward search window in the backwardfield or frame by accumulating the errors between such pixels whicharrive at its two inputs, and means for storing the partial results inorder to arrive at a total error for each block; detection meansconnected to said processing elements for selecting a motion vectorrelated to the location of the best matching pair of blocks from theforward field or frame and the backward field or frame, in order toselect one motion vector for the current of said notional pixel blocks(I FB), whereby a first half of said motion vector points to thelocation of the best matching block in the backward field or framesearch window and the second half of said motion vector points to thelocation of the best matching block in the forward field or frame searchwindow, both vector parts depending on a same candidate motion vector,wherein the candidate motion vector all pass through the same points insaid current field or frame to be interpolated, and wherein every pointin the interpolated field has precisely one motion vector assignedthereto to eliminate motion vector gaps and conflicts within areas inthe interpolate field.
 2. Apparatus according to claim 1, wherein saidselection means round the first half and the second half of said motionvector in such a way that a related sum remains correct.
 3. Method formotion estimation using two-sided block matching wherein a current fieldor frame to be interpolated and which is non-existing in an originalpicture signal is regarded as having notional pixel blocks which coverthe field or frame to be interpolated and are correctly placed in thatfield or frame to be interpolated and wherein respective forwardcandidate pixel blocks within a forward search window in a forward fieldor frame are matched with respective backward candidate pixel blockswithin a backward search window in a backward field or frame in order toselect one motion vector for a current of said notional pixel blockscomprising the step of:the two sided block matching is performed so thata first part of said motion vector for a current of said notional pixelblocks points to the location of the best matching block out of thebackward field or frame search window and the other part of said motionvector points to the location of the best matching block out of theforward field or frame search window, both vector parts depending on asame candidate motion vector, wherein the candidate motion vectors allpass through the same points in said current field or frame to beinterpolated, and wherein every point in the interpolated field hasprecisely one motion vector assigned thereto to eliminate motion vectorgaps and conflicts within areas in the interpolated field.
 4. Methodaccording to claim 3, wherein the first part and the other part of saidmotion vector are each one half of the motion vector.
 5. Methodaccording to claim 4, wherein the first half and the second half of saidmotion vector are rounded in such a way that a related sum remainscorrect.
 6. Method according to claim 3, wherein said block matching insaid backward and said forward field or frame is performed usinginterpolated half-pixel values.
 7. Method according to claim 3, whereinthis method is used for field or frame rate upconversion, slow motiongeneration, noise reduction or for controlling displays.